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ABSTRACT 

This paper presents findings of a study that examined 
the effectiveness of the guidance program used by change agents in 
the Dutch School Improvement Project (NSIP) on program 
implementation. Sixteen schools composed the experimental group and 
13 made up the control group. The experimental variabl-e was the 
guidance strategy of the NSIP. The two effect variables included: (1) 
the degree to which teachers increased the available task-oriented 
instruction and learning time; and (2) the effective organization of 
preventive care for low-achieving students. Data were gathered 
through a comparison of the program's planning data with the change 
agents* execution data, classroom observations, and a questionnaire. 
Findings indicate that during the program's first year of adoption, 
the improvement strategy has been implemented, though insufficiently. 
Schools in the experimental group demonstrated a slight growth in 
task-oriented learning time; the control group exhibited no growth. 
Finally, the experimental group showed an increase in joint planning 
and remedial activities. A conclusion is that the program is off to a 
good start. (LMI) 
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Evaluatioa questions ^ 

The following questions are central to this evaluation research: 

1) Does the quidance programme used by change agents in the National School Improvement Project 
lead to implementation of the school improvement programme? 

2) Does instruction in which the programme has been adequately opeiitionalized lead to the desired 
(cognitive and affective) learning outcomes of pupils? 

The second research question, as represented above, has been formulated in conditional form: only if the 
school improvement programme has been adequately implemented can research take place into the effect of 
teacher behaviour on learner performance. 

Research design 

We planned two experiments. 

The first experiment is aimed at determining the effectiveness of the guidance programme used by change 
agents in the National School Improvement Project in implementing the school improvement programme. 

The second experiment will concern the second research question: Does the adequately implemented school 
improvement programme lead to the desired learning outcomes? 

In this paper only the design of the first experiment is discussed, since the second experiment was (of 
course) not yet scheduled for the school year 1991-1992. 

In order to answer our first research question, concerning the effectiveness of the guidance programme for 
the implementation of the school improvement programme, we chose the interrupted time series design with 
a nonequivalent no-treatment control group time series (cf. Cook & Campbell, 1979). The measurement 
design is shown in the diagram immediately below. 

01 Xl 02 X2 03 X3 04 X5 05 X6 06 O? 

02 O2 O3 O4 O5 06 O7 

The measurements were and will be taken in the months of January/February and May/June (Oi = Jan/Feb 
1992, O2 = May/June 1992, O3 = Jan/Feb 1993, O4 = May/June 1993, Os = Jan/Feb 1994, 06 = May/June 
1994). The permanence of the effect will be tested by means of a postmeasurcment (0? = May/June 1995) 

At this moment we (of course) can only discuss the first two measurements. So the measurement design for 
the situation at this moment looks as follows. 

Ol Xl O2 
Ol O2 

The experimental group was given the treatment, the control group received no form of treatment. 

Each measurement consists of two observations of a reading lesson and a the completion of a questionnaire 
by the teachers. The observations take place in intervals of approximately four weeks. 

Twenty-nine schools in all were involved in the research, sixteen in the experimental group and thirteen in 
the control group. 
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Dependent/ effect variables 



The experimental variable in the experiment is the guidance strategy of the National School Improvement 
Project 

The experiment contains several effect variables. We will concentrate in this paper on two of them. The first 
variable relates to thft degree to which teachers succeed in increasing the available task-oriented instruction 
and learning time. The second group of these variables concerns the effective organization of preventive care 
for pupils with poor results. 

Degree of Implementation 

The implementation of the school improvement programme can be operationaliscd on the basis of the change 
agent activities. 

We made detailed charts of the guidance activities of the change agents with the, aid of a monitoring system 
(Timmermans, 1990). The monitoring system is a means of process control which consists of logbook 
registration forms concerning the planning, aims and execution of the guidance activities. The change agents 
complete these forms after each guidance contact. By comparing the execution data with the planning data a 
picture can be obtained of the guidance interventions of each change agent. 

Planning data 

The planning data are stated in two documents: the general task commitment and the project script 

The general task commitment specifies that change agents should invest one half day per 14 days in the 

NSP. In addition change agents construct a guidance plan for each participating school. 

The project script contains the activities that change agents undertake on two levels, namely the team level 
and the level of the individual teacher. This includes guidance meetings organized for teachers on the 
teamlevel and classroom consultations with teachers. The guidance meetings arc organized on the basis of 
the agendas provided by the project leaders. Although the change agents are supposed to follow the agendas 
in organizing guidance meetings, it is very well possible that after some time the meetings assume a more 
school-specific character, not every school has the same point of departure or the same need for certain 
forms of information. Change agents can therefore vary the form of the meetings at their individual school 
and can have certain subjects discussed to various degrees of depth. However, they are required to bring up 
for discussion all the essential parts of an agenda. 

Execution data 

In order to determine how far the intended guidance activities have in fact been executed, change agents are 
asked to produce detailed reports of all their activities. The contents of these reports arc subsequently 
quantified to be able to examine, first, the degree of correspondence between the planned guidance activities 
and the actual guidance activities and, secondly, the possible differences between change agents in the degree 
of actual guidance and whether these differences lead to differences in project implementation or teacher 
behaviour. 

We now give a summary of the amount and the themes of guidance between January 1992 and June 1992. 
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summary of amount and themes of guidance between January 1992 and June 1992 




fitequency of 




theme 


school 


team 

consultati- 
on 


classroom 
consul- 
tation 


coaching 

individual 

teacher 


amount of 
hours 


learning 
time 


preventive 
care 


a 


9 


1 


7 


2.5 


A- 


+ 


b 


7 


2 




2 


+ 


+ 


c 


7 






2 


+ 


+ 


d 


6 


2 


5 


2 




+ 


e 


3 




3 


4.5 






f 


8 




4 


4 


+ 


+ 


g 


7 




1 


1 


+ 


+ 


h 


6 




1 


1 


+ 


+ 


i 


8 




1 


2.5 


+ 


+ 


j 


15 


- 


2 


2 


+ 


+ 


k 


11 




2 


2 


+ 


+ 


1 


9 




1 


6 


+ 


+ 


m 


10 




1 


6 


+ 


-/+ 


n 


6 


1 




6 


+ 


-/+ 


0 


10 




3 


11.5 


+ 


-/+ 


P 




5 


4 




+ 





From the table above we have to conclude that this first project year was mostly an adoption year. This is 
clearly indicated by the guidance activities that were carried out. At most schools a great deal of time was 
spent in team consultation on clarifying the initial stage of the project. As a result classroom consultation and 
coaching of individual teachers was started somewhat late: too little classroom consultations have taken 
place. 

The National Schoolimprovement script is to a large extent being followed by all change agents. But not all 
teachers have received approximately the same treatment. None of the teachers have had enough treatment 
yet In other words the improvement strategy has been implemented, but not sufficiently yet. 

Effects 

The degree of execution of prescribed activities for teachers is measured by means of an observation 
instrument and a questionnaire. 

the observation instrument 

The effective learning and instruction time is determined by means of observations. The procedure goes as 
follows: During twenty minutes of a reading lesson the amount of effective time 2»pent '^y teachers and pupils 
is charted by means of observations with a time sampling instrument (Veenman et al., 1988). This instrument 
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gives an estimate of the task-oriented learning time of learners. For obtaining a score for each class at least 
two measurements are made. The mean score becomes the score of each class. The mean scores of these 
class means for the experimental group and the control grovp are presented in the next table. 



Task-oriented learning time of pupils 




January 1992 


June 1992 




exp. group 


contr. 
group. 


exp. group 


contr. group. 


task-oriented learning time 


703% 


75.7% 


72.1% 


74.6% 



We can conclude that there is a slight growth in the experimental group, while there is no growth in the 
control group. 



the questionaire 

The project uses a model of preventive care for pupils with poor results as developed by Kool & Van der 
Lcij (1985). In this model procedures for information processhig and planning are integrated, as are the 
various levels wich can be distingdshed within the school, namely school level, class level and pupil level. 
This model formed the basis for the construction of a set of instruments for the measuring of joint planning. 
Scales were constructed for each of the stages that Kool & Van der Leij distinguish in their model. 

The original model needed adjusting in two areas. There is a strong emphasis in the project on a wider range 
of preventive care. As a result the class level receives the most attention. This emphasis should obviously be 
reflected in our instruments. In addition, one of the first matters to be attended to in the fiamework of joint 
planning is the registration of the aim of school improvement in terms of desired learning outcomes. Our set 
of instruments contains specially constructed scales for this purpose. 

The names of the scales and the mean scores are represented in the table immediately below. 
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Instruments for measuring preventive care 




January 1992 


June 1992 




exo, fitrouD 


contr. 
group. 


exp. eroup 


contr. group. 




mean (s.d.) 


mean (s.d.) 


mean (s.d.) 


mean (s.d.) 


Setting aims: 










Setting targets 


.58 (.33) 


.57 (.31) 


.63 (.33) 


.56 (.31) 


Degree of completion of set targets 


.49 (.27) 


.47 (.23) 


.50 (.26) 


.41 (.25) 


Signalling: 










Degree to which problems of pq^ils are 
signalled 


.95 (.14) 


.97 (.08) 


1.00 (.00) 


.95 (.11) 


Analyzing/diagnosing: 










Degree to which learning results are related 
to the instruction given 


.72 (.15) 


.62 (.25) 


.76 (.23) 


.56 (.24) 


Degree to which the teachers themselves 
diagnose pupils with poor results 


.59 (.23) 


.73 (.16) 


.59 (.24) 


.73 (.19) 


Carrying out and evaluating results: 










Degree to which activities are recorded in a 
group plan 


"XA ( AON 


1 Q / 
.lO 1 ) 






Degree to which p\q)ils with poor results 
are taken care of as a group 


.56 (.16) 


.50 (.14) 


.55 (.12) 


.49 (,19) 


Degree to which activities are recorded in a 
plan for remedial activities 


.49 (.48) 


.36 (.48) 


.70 (.43) 


.21 (.43) 


Degree to which the teacher gives adjusted 
instruction 


.68 (.15) 


.65 (.10) 


.68 (.13) 


.66 (.11) 


Degree to which joint planning is a matter 
for the school team 


.46 (.31) 


.40 (.18) 


.61 (.25) 


.32 (.18) 



We can conclude from the table above that in six out of ten measures there is growth in the mean of the 
experimental gtovp while the mean of the control group remains more or les constant In partivular, the 
degree to which activities are recorded in a groi^ plan and in a plan for remedial activities has increased 
considerably. Furthermore, there is an increase in the degree to which joint planning is a matter for the 
school team. The means of the other four measures remain constant for the experimental groi^ till now. 



Conclusions 

The data that have been gathered allow us to conclude that the project is off to a good start. Implementation 
has started and the growth on the effect variables has started for the experimental group, while there is no 
growth for the control group. For the first project year, this is a good result. 
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